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EXECUTIVE SUMMARY 

This document presents findings based on a third-year evaluation of Trenholm State (AL) 
Technical College’s National Aeronautics and Space Administration (NASA) - supported 
High School Science Enrichment Program (HSSEP). HSSEP is an external (to school) 
program for area students from groups that are underrepresented in the mathematics, 
science, engineering and technology (MSET) professions. In addition to gaining insight into 
scientific careers, HSSEP participants learn about and deliver presentations that focus on 
mathematics applications, scientific problem-solving and computer programming during a 
seven-week summer or 10-week Academic-Year Saturday session. 

The purpose of this follow-up evaluation report is three-fold: 

1) to determine whether HSSEP has been able to sustain and improve upon its initial 
success, despite increased enrollment and the inclusion of more students from at-risk 
groups (e.g., boys, rural students); 

2) to determine whether second-year improvements in participants' math/science; 
classroom performance continues; and 

3) to determine whether there are any correlates of success that can be used to help 
HSSEP increase the diversity in its student population while continuing to produce 
positive participant outcomes . 


Using content analyses of existing program documents and student records; on-site 
observations and interviews; and surveys that provide implementation assessment and self- 
reported outcome data, the author found that: 

HSSEP has been able to: 

• serve increased numbers of students without benefit of additional funding; 

• increase the percentage of students in each cohort who are 9' h graders; and 

• significantly increase the participation of boys over the last two years. 

Educational services were rated “Very Effective" by at least half - and as many as 88 percent of 
the participants. Moreover, with the exception of field trips, all of the Program services were 
perceived as being more effectively delivered than in 1995-’96. 

HSSEP’s frequent student/parent/staff interactions and its emphasis on rigorous participation 
requirements, parental involvement in decision-making and continuous implementation 
improvement have increased attendance and retention. The significantly higher retention of 
freshmen (versus sophomores) lends additional support to the Program’s decision to shift its 
recruiting focus to ninth graders. Student interviewees indicate that earlier participation: 

• helps them avoid the initial failure that caused siblings/peers to drop out of the 
math/science pipeline; 

• unlearn previously learned errors; 

• develop the practice of working in groups; and 

• gain opportunities to earn money as tutors -- while reinforcing skills that they might 
otherwise have forgotten. 
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HSSEP continues to make a significant, positive difference in the perceived MSET 
educational outcomes of its participants. At least half of all respondents strongly agree that 

HSSEP had a positive impact on their MSET outcomes. 

Given concerns about the extent to which HSSEP experiences are translated into improved 
classroom performance, input from participants’ classroom teachers was elicited for the first time. 
Teachers' overall ratings were very similar to students self-assessments. While 51 percent of 
students indicated that there math, science and computing was “very much” improved, 45 percent 
of the students rated by classroom teachers received a comparable rating. 

Among the improvements most frequently cited in teachers’ essay responses were: 


• Greater interest/motivation; 

• More confidence; 

• More focus and seriousness in approaching work; and 

• Better preparation and attention to homework 

Participant improvement was less evident when other outcome measures are considered. 

HSSEP students' ACT mathematics scores continued to reflect the substantial or statistically 
significant pre-post improvement noted in the 1 995-’96 Evaluation Report. In contrast, last year’s 
gains in the science area were not replicated. 

Previously expressed concerns (see First Annual Report) about concept 
mastery and educational standards surfaced again when analyses of 
math/science GPAs for 1996-97 participants revealed substantially 
weaker performance in both math and science. Not only did the 
establishment of closer relationships with local school districts and efforts 
to achieve more compatibility between HSSEP and classroom curricula 
fail to sustain the improvement noted last year, but the performance of 
students with sustained participation in HSSEP also declined. 

Follow-up interviews and secondary analyses of project documentation and student performance 
data suggest that participants’ classroom performance is, in part, due to the inclusion of more “at- 
risk” students, policies that permit participation based on interest rather than performance and 
less-than-desired parental participation. For example. 


• As a result of more aggressive outreach, the number of boys recruited by HSSEP 
increased significantly. Given their poorer classroom performance, boys contributed 
to the overall decline in HSSEP students’ math/ science GPAs; 

• Students’ who were retained despite ACT pre-post scores that did not improve, were 
more likely to be poor classroom performers (i.e., those with math/science GPAs 
below 2.5); 

• Among Persisters, those with the highest percentage of possible sessions attended 
were better performers in the classroom; and 

• Those whose parents were actively involved in HSSEP were more likely to perform 
well in the classroom. 

Post-high school outcomes for the 1997 HSSEP class are positive, but not quite as good as in 
the past two years. Although there was a 100 percent graduation rate among HSSEP students, 
enrollment in college MSET majors was only 50 percent versus 65 percent in 1995 and 78 
percent for the 1996 graduates. 
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As noted in the January 1996 HSSEP Evaluation Report, HSSEP alumni’s post-high school 
outcomes are in stark contrast to those of their Montgomery County counterparts. 
Countywide, only 6.5 percent of African-Americans completed special academic programs in 
science and engineering and only 9 percent planned to pursue careers in engineering, 
mathematics or science. 

Based on findings presented herein, the Evaluator recommends that: 

1) In light of the increased emphasis on recruiting boys, HSSEP should make provisions 
for identifying the unique needs, learning influences and preferences that may need to 
be considered in order to successfully intervene on behalf of a population that is 
extremely “at-risk” in the Montgomery/Lowndes County area; 

2) The study skills/test-taking course receive special attention in next year’s evaluation, 
so as to determine the reason why classroom teachers continue to cite students’ 
shortcomings in this area; 

3) HSSEP consider meeting its reading, writing and speaking improvement objectives by 
integrating communications activities into MSET classes. This will allow the Program 
to use time currently devoted to the Communications Component for additional 
MSET intervention; 

4) HSSEP use measures of Program and classroom performance to determine eligibility 
for continued Program participation, receipt of stipend and other incentives that might 
address what may be diminished motivation among students (e.g., 3-4 sessions 
attended) who are neither new to the Program or seriously committed to preparing for 
an MSET career; and 

5) Strategies be developed to increase the level of participation among parents. As a 
first step, the results of this Report should be shared with parents in a “town meeting” 
format that gives them an opportunity to develop a sense of ownership of the program 
and empowerment, in terms of influencing their children’s educational outcomes. 
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OVERVIEW 


PROGRAM RATIONALE AND HISTORY 

This document presents findings based on a third-year evaluation of Trenholm State (AL) 
Technical College's National Aeronautics and Space Administration (NASA) - supported 
High School Science Enrichment Program (HSSEP). HSSEP is an external (to school) 
program that provides math, science, computer science and communications enrichment 
for high school students who reside in Montgomery or Lowndes County (AL). HSSEP’s 
proposal called for the Program to serve 40-50 students from groups that are 
underrepresented in the mathematics, science, engineering and technology (MSET) 
professions. HSSEP Participants have diverse academic achievement histories, but share an 
expressed interest in mathematics and science, a willingness to regularly attend a seven- 
week summer or 10-week Saturday session during the academic year. 

Previous HSSEP evaluation reports contain detailed information regarding the curriculum, 
whose components include: 

Mathematics Applications.- Students use graphing calculators and a variety of 
software packages in math and statistics to enhance their understanding of concepts taught in 
the Integrated Science course and prepare them to improve their performance on 
standardized aptitude and achievement tests. 

Integrated Science.- Students are introduced to the scientific method and the 
integration of problem-solving skills in science through the use of hands-on laboratory 
experiments, demonstrations, audio visuals and group projects. 

Computer Literacy and Programming.- Students learn the basic terminology of 
computers, computer operations, programming using BASIC and Turbo PASCAL, and 
spreadsheet, data base and word processing software applications. 

Communications.- Students prepare and deliver presentations on topics in applied 
math and science to enrich reading, writing, comprehension, listening and speaking skills. 

Field Experiences in Science.- Activities include field trips and seminars, career 
interest inventory completion and analysis; presentations from role models; and career 
games. 


HSSEP’s previous evaluation reports indicate that after some initial start-up difficulties, 
HSSEP was being fully and effectively implemented. Interim measures (e.g., participants’ 
enrollment in challenging math/science courses, attendance/persistence rates, indications of 
increased interest in math/science) and post-high school outcomes (e.g., college enrollment, 
choice of MSET major) for initial HSSEP graduates suggest that HSSEP was making 
progress in its efforts to provide access to MSET careers for historically by-passed students. 


EVALUATION RATIONALE 

The purpose of this follow-up evaluation report is three-fold: 

1) to determine whether HSSEP has been able to sustain and improve upon its initial 
success, despite increased enrollment and the inclusion of more students from at-risk 
groups (e.g., boys, rural students); 
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2) to determine whether second-year improvements in participants' math/science; 
classroom performance continues; and 

3) to determine whether there are any correlates of success that can be used to help 
HSSEP increase the diversity in its student population while continuing to produce 
positive participant outcomes . 

METHQDQLQQY 

Data produced by the HSSEP information system and evaluation design are derived from: 

• content analyses of existing program documents (e.g., previously submitted 
evaluation reports, participation records); 

• on-site observations and interviews with Trenholm administrators, HSSEP 
administrators, faculty and students; 

• participant surveys that provide implementation assessment and self-reported 
outcome data; 

• Surveys assessing student performance and indicating areas needing intervention 
that were completed by participants’ classroom teachers 

• analyses of students’ transcripts; and 

• Participants’ scores on pre-post administrations of parallel forms of the American 
College Test mathematics and science practice examinations. 

Descriptive and comparative analyses of the above-referenced data provide the basis for the 

findings and discussion presented in the next section 
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FINDINGS AND DISCUSSION 


PROGRAM IMPLEMENTATION 
Recruitment 

HSSEP’s annual enrollment goal is 40-50 students from groups underrepresented in 
mathematics, science, engineering and technology professions (MSET). In addition, earlier 
evaluations suggested two other recruiting goals: (1) increased representation of 9 th graders who 
have more time (prior to graduating) in which to benefit from educational intervention; and (2) 
increased participation by boys whose high school performance and college enrollment 
consistently lag behind those of girls in Montgomery and Lowndes County. The data in Table 1 
indicate that HSSEP is achieving all of its recruiting goals. Specifically, HSSEP has been able to: 

• serve increased numbers of students without benefit of additional funding; 

• increase the percentage of students in each cohort who are 9 lh graders; and 

• significantly increase the participation of boys over the last two years. 


TABLE 1 


The Distribution of HSSEP Students By Cohort, Gander and Grade 

New Admit 


Cohort 

# New 
Admits 

Male 

Female 

9 

Grade Level 

10 n 

12 

# Returnees 

Total 

1994-‘95 

65 

21 

44 

13 

25 

8 

19 

N/A 

65 

1995-‘96 

49 

8 

41 

21 

20 

8 

0 

25 

74 

1996-‘97 

47 

21 

26 

34 

13' 

0 

0 

35 

82 

1997-‘98 
Summer Only 

15 

8 

7 

15 

0 

0 

0 

43 

58 


Providing Educational Intervention Services 

Participating students’ survey responses indicate that HSSEP was fully implemented. Nearly all 
respondents indicated that they were exposed to each of the Program components described in 
HSSEP’s original proposal or subsequent modifications of that technical approach. In addition to 
being fully implemented, comparisons of ratings by 1995-’96 and 1996-’97 participants suggest 
that the delivery of HSSEP services is improving as the staff and its director gain experience and 
modify the approach to meet emerging needs and concerns. 

The data presented in Table 2 summarize students’ perceptions of the quality of services they 
received. Those data reveal that all of the educational services were rated “Very Effective” by at 
least half - and as many as 88 percent of the participants. Moreover, with the exception of field 
trips, all of the Program services were perceived as being more effectively delivered than in 1995- 
’96. 
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TABLE 2 


Percentage of Students Rating HSSEP Service Delivery “Very Effective” 


ERVICE 1995-*96 1996-‘97 


Orientation clearly spelled out expectations 70 83 

Faculty took the time to answer questions 78 88 

Laboratory and classroom facilities 51 70 

Field trips provided good career information 71 67 

Activities showing parents how to be more supportive 41 66 

Minority role models provided E&S career insights 55 60 

Staff were available to discuss my concerns 50 57 

Teacher preparation and knowledge 61 79 

Strategies for involving all students 53 54 

Provided adequate individual attention 54 65 

Provided competent Teaching Assistants 58 84 

Allowed sufficient time to complete course requirements 20 ■ 49 


The Focus of Future Interventions 

HSSEP participants' classroom teachers were asked to indicate areas in which these students 
needed additional intervention. The most frequent suggestions and the percentage of responses 
citing the need were: 

• Tutoring in specific subjects (19%) 

• Test-taking skills (17%) 

• Study skills (14%) 

Depending on the classroom performance of HSSEP participants, Program administrators may 
wish to consider modifying the intervention to incorporate or increase the time devoted to the 
above teacher recommendations. 
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PROGRAM OUTCOMES 

Attendance and Retention of Students in the MSET Pipeline 


Attrition in educational enrichment programs often becomes a problem at the high school level. 
Student participation declines either because the program has ceased to meet their needs or 
because other commitments take precedence. In contrast, data suggest that HSSEP's frequent 
student/parent/staff interactions and its emphasis on rigorous participation requirements, 
parental involvement in decision-making and continuous implementation improvement have 
increased student participation. Attendance continues to be in the mid-nineties with very few 
students dropping out during any given session. In addition, Table 3 indicates that the 
percentage of HSSEP students who forego other interests to return for additional MSET 
enrichment continues to grow. 


TABLE 3 

Program Persistence Rate by Grade 



Cohort 

9 lh Grade 
Admits 

1 0 th Grade 
Admits 


1994-95 

38.5 

15.4 


1995-96 

68.4 

18.2 


1996-97 

72.7 

37.9 


The increasing number of HSSEP students remaining in the MSET pipeline is consistent with the 
perceived benefits (among students) of Program participation reported in the previous section. 
The statistically significant (p<.01) differences in the persistence of those who enter HSSEP as 
freshmen (versus sophomores) lends additional support to the Program’s decision to shift its 
recruiting focus to ninth graders. As reported previously, student interviewees indicate that 
earlier participation: 


• helps them avoid the initial failure that caused siblings/peers to drop out of the 
math/science pipeline; 

• unlearn previously learned errors; 

• develop the practice of working in groups; and 

• gain opportunities to earn money as tutors — while reinforcing skills that they might 
otherwise have forgotten. 
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Indications of HSSEP’s Positive Participant Impact 

There are numerous measures by which one can assess the extent to which HSSEP is helping 
participants achieve the Program’s objectives of increased math/science interest and 
proficiency. Analyses of these measures (e.g., participant self-reports; ACT pretest-posttest 
scores) reveal that HSSEP continues to make a significant, positive difference in the MSET 
educational outcomes of its participants. Unlike last year however, there are some indications 
that additional modifications are needed to meet the needs of a changing HSSEP participant 
population. 

Participants’ Personal Assessments 

Students' survey responses reveal their belief that HSSEP is not only effectively implemented, 
but also plays a major role in increasing their math/science/technology interest and learning. 
Table 4 indicates that at least half of all respondents strongly agree that HSSEP had a 
positive impact on their MSET outcomes. Comparisons of 1995-’96 and 1996-’97 participant 
assessments indicate that the communications component was somewhat more successful in 
improving students’ writing and speaking skills. On the other hand, while there are some small 
year-to-year fluctuations in the Program’s effect on participants’ MSET skills, the overall picture 
remains largely unchanged. 


TABLE 4 

Percentage of Students Who Strongly Agree With Statements Concerning HSSEP’s 
Positive Impact on Their MSET Interest and Learning 


Student Outcome 

1995-96 

1996-97 

HSSEP increased my confidence that 1 can do well in school 

65 

60 

1 would like to continue participating in HSSEP 

59 

63 

1 learned a lot about science 

61 

63 

1 learned a lot about mathematics 

43 

44 

1 learned a lot about computer programming 

46 

51 

1 became more interested in computer programming 

63 

59 

1 became more interested in science 

50 

49 

1 became more interested in math 

31 

37 

My writing skills improved 

39 

47 

My reading skills improved 

44 

42 

My public speaking skills improved 

42 

55 
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Given the weak performance of Alabama’s African-American students on the National 
Assessment of Educational Progress’ (NAEP) 8 1 * Grade Mathematics Test, HSSEP should 
consider examining why its positive impact is least pronounced in the mathematics area. 
Administrators may also want to consider whether, in light of HSSEP’s modest communications 
success, students might benefit from greater exposure to MSET experiences. 


Classroom Teachers’ Assessments of HSSEP Students’ MSET Progress 

Given concerns about the extent to which HSSEP experiences are translated into improved 
classroom performance, input from participants’ classroom teachers was elicited for the first time. 
Teachers’ overall ratings were very similar to students self-assessments. While 51 percent of 
students indicated that there math, science and computing was “very much” improved, 45 percent 
of the students rated by classroom teachers received a comparable rating (see Table 5). 


TABLE 5 


Classroom Teachers’ Assessments of HSSEP Students’ MSET Improvement 


Category 

Sustained/Enhanced Excellence 
Great Improvement 
Some Improvement 
The Same/Diminished Performance 


. Percent of Students 
08 
37 
39 
16 


Among the improvements most frequently cited in teachers' essay responses were: 

• Greater interest/motivation; 

• More confidence; 

• More focus and seriousness in approaching work; and 

• Better preparation and attention to homework 


ACT Test Score Improvement 

Beginning in the 1996 summer session, parallel forms of a practice ACT exam were 
administered on a pre-test post-test basis. Score changes measured how much students 
learned in a preparation course that offered: test-taking tips; sample problems; cognitive skill 
development exercises; insight into the psychometric principles underlying test construction; and 
other activities with demonstrated effectiveness in raising students' aptitude test scores. 
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HSSEP students’ ACT mathematics scores continued to reflect the substantial or statistically 
significant pre-post improvement noted in the 1995-’96 Evaluation Report. In contrast, last year’s 
gains in the science area were not replicated (see Table 6). 


TABLE 6 

Mean ACT Pre and Post-Test Math/Science Scores (By Grade and Year) 

Grade 



2 


10 

11 


12 



1995-96 

96-97 

95-96 

96-97 

95-96 

96-97 

95-96 

96-97 

Math Pre-Test 

11.0 

11.2 

13.9 

13.1 

10.7 

12.6 

11.5 

8.0 

Math Post -Test 

14.4 

15.5 

17.7 

16.9 

15.4 

14.8 

15.0 

13.3 

Science Pre-Test 

18.7 

17.5 

15.4 

16.3 

17.6 

16.3 

16.0 

22.8 

Science Post-Test 

17.2 

15.6 

18.3 

16.9 

20.0 

16.6 

24.8 

23.8 


Because the critical mass of HSSEP students has yet to have the ACT administered for college 
admission purposes, it is still too early to determine the impact of test preparation activities on 
scores that are meaningful. In fact, the Director has speculated that test performance is 
negatively impacted by multiple administrations to students participating in HSSEP for a second 
year. While the reasons for the (relatively) poorer pre-post performance are unclear, the 
relationship between pre-post test differences and student performance will be explored in a 
subsequent section. 


Math/Science Grades 

The January 1996 Evaluation Report noted that many HSSEP students would be able to take 
calculus in high school — because of taking math courses above their grade level in junior 
high school. While acknowledging the potential benefits of this trend, the Report raised a 
concern about students' lower-than-expected math/science grades. It was hypothesized that 
HSSEP students, in their haste to take advanced courses, might not be (1) meeting the 
performance standards against which they are graded; or (2) mastering concepts critical to 
their future math/science success. 

The concern about concept mastery and educational standards surfaced again when analyses of 
math/science GPAs for 1996-97 participants revealed that substantially weaker performance in 
both math and science. Not only did the establishment of closer relationships with local school 
districts and efforts to achieve more compatibility between HSSEP and classroom curricula fail to 
sustain the improvement noted last year, but the performance of students with sustained 
participation in HSSEP also declined (see below) 
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GROUP 


GPA 


1994-‘95 Persisters 

2.87 

1995-‘96 Persisters 

3.25 

1996-’97 Persisters 

2.75 


In an effort to explain the sudden emergence of poor classroom performance, the relationship 
between previously cited program changes and math/science GPA were examined. While small 
sample sizes and other methodological considerations limit our ability to quantify and generalize, 
from analyses of the available data, the following relationships appear to account for some of the 
performance differences noted among HSSEP students: 

• Because of their underrepresentation in previous HSSEP cohorts, the number of 
boys recruited by the Program increased significantly. Given their poorer classroom 
performance, boys contributed to the overall decline in HSSEP students' math/ 
science GPAs; 

• Students' whose ACT pre-post scores did not improve, were more likely to be poor 
classroom performers (i.e., those with math/science GPAs below 2.5); 

• Among Persisters, those with the highest percentage of possible sessions attended 
were better performers in the classroom; and 

• Those whose parents were actively involved in HSSEP were more likely to perform 
well in the classroom. 


1997 Graduates’ Post-Proaram Outcomes 

Post-high school outcomes for the 1997 HSSEP class are positive, but not quite as good as in 
the past two years. Although there was a 100 percent graduation rate among HSSEP students, 
enrollment in college MSET majors was only 50 percent versus 65 percent in 1995 and 78 
percent for the 1996 graduates. Given the small sample size fluctuations of this magnitude are 
not unexpected. 

As noted in the January 1996 HSSEP Evaluation Report, HSSEP alumni’s post-high school 
outcomes are in stark contrast to those of their Montgomery County counterparts. 
Countywide, only 6.5 percent of African-Americans completed special academic programs in 
science and engineering and only 9 percent planned to pursue careers in engineering, 
mathematics or science. 
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RECOMMENDATIONS 


Based on findings presented herein, the Evaluator recommends that: 

1) In light of the increased emphasis on recruiting boys, HSSEP should make provisions 
for identifying the unique needs, learning influences and preferences that may need to 
be considered in order to successfully intervene on behalf of a population that is 
extremely “at-risk” in the Montgomery/Lowndes County area; 

2) The study skills/test-taking course receive special attention in next year's evaluation, 
so as to determine the reason why classroom teachers continue to cite students’ 
shortcomings in this area; 

3) HSSEP consider meeting its reading, writing and speaking improvement objectives by 
integrating communications activities into MSET classes. This will allow the Program 
to use time currently devoted to the Communications Component for additional 
MSET intervention; 

4) HSSEP use measures of Program and classroom performance to determine eligibility 
for continued Program participation, receipt of stipend and other incentives that might 
address what may be diminished motivation among students (e.g., 3-4 sessions 
attended) who are neither new to the Program or seriously committed to preparing for 
an MSET career; and 

5) Strategies be developed to increase the level of participation among parents. As a 
first step, the results of this Report should be shared with parents in a “town meeting” 
format that gives them an opportunity to develop a sense of ownership of the program 
and empowerment, in terms of influencing their children’s educational outcomes. 
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